FCPlanner: A Planning Strategy for First-Order MDPs

States We formalize abstract states symbolically, within the Fluent Calculus (FC) (Hölldobler & Schneeberger 1990). Fluent Calculus, much like Situation Calculus, is a logical approach to modelling dynamically changing systems based on firstorder logic. One could indeed argue that Fluent Calculus and Situation Calculus have very much in common. But the latter has the following disadvantage: Knowledge of the current state is represented indirectly via the initial conditions and the actions which the agent has performed up to a point. As a consequence, each time a condition is evaluated in an agent program, the entire history of actions is involved in the computation. This requires ever increasing computational effort as the agent proceeds, so that this concept does not scale up well to long-term agent control (Thielscher 2004). Fluent Calculus overcomes the aforementioned unfolding problem by providing the crucial concept of an explicit state representation. The information on what is true in the current state of the world is effortlessly extracted from the state description without tracing back to the initial state. Therefore we have opted for Fluent Calculus as logical formalism ∗Supported by the research training group GRK 334/3 (DFG). Corresponding author. underlying our automated symbolic dynamic programming approach. In FC, functions whose values vary from state to state are called fluents and are denoted by function symbols. For example, the fluent on(X, table) denotes the presence of a block X on the table. A state is a multiset of fluents represented as a term, called fluent term, using a constant 1 denoting the empty multiset and a binary function symbol ◦ denoting multiset union that is associative, commutative and admits unit element. For example, a state in which the block a is on the block b and b is on the table is specified by on(a, b) ◦ on(b, table). Constants are denoted by small letters, variables by capital ones and substitutions by θ or σ. Abstract states are characterized by means of conditions that must hold in each ground instance thereof and, thus, they represent sets of real-world states. Informally, abstract states can be specified by stating that particular fluent terms do or do not hold. We refer to such abstract states as CN-states, where C stands for conjunction and N for negation, respectively. Formally, let L be a set of fluent terms. A CN-state is a pair (P,N ), where P ∈ L, N ∈ 2L. Let · be a mapping from fluent terms to multisets of fluents, which can be formally defined as follows: 1 = {̇}̇ or F = {̇F }̇, if F is a fluent, or (F ◦G) = F ∪̇ G , where F,G are fluent terms and ∪̇ is a multiset union. Let I = (∆, ·I) be an interpretation, whose domain ∆ is the set of all finite multisets of ground fluents and every CN-state Z = (P,N ) is mapped ontostates are characterized by means of conditions that must hold in each ground instance thereof and, thus, they represent sets of real-world states. Informally, abstract states can be specified by stating that particular fluent terms do or do not hold. We refer to such abstract states as CN-states, where C stands for conjunction and N for negation, respectively. Formally, let L be a set of fluent terms. A CN-state is a pair (P,N ), where P ∈ L, N ∈ 2L. Let · be a mapping from fluent terms to multisets of fluents, which can be formally defined as follows: 1 = {̇}̇ or F = {̇F }̇, if F is a fluent, or (F ◦G) = F ∪̇ G , where F,G are fluent terms and ∪̇ is a multiset union. Let I = (∆, ·I) be an interpretation, whose domain ∆ is the set of all finite multisets of ground fluents and every CN-state Z = (P,N ) is mapped onto ZI = {d ∈ ∆ | ∃θ. (Pθ) . ⊆ d ∧ ∀N ∈ N .∀σ.((Nθ)σ) . * d} , where . ⊆ is a submultiset relation. In other words, the P -part of a state Z describes properties that a real-world state should satisfy, whereas N part specifies the properties that are not allowed to fulfil. For example, the CN-state Z = (on(X, table) ◦ red(X), {on(Y,X)}) represents all states in which there exists a red object that is on the table and clear, viz., none of other objects covers it. Thus, the real-world state z = . {on(a, table), red(a), on(b, table), green(b) . } is specified by Z. Whereas, z′ = . {on(a, table), red(a), on(b, a) . }